Create a parallel abstract class hierarchy for JDBC vs non-JDBC classes #24172

akashkulk · 2023-03-16T23:04:22Z

This PR is part of #23808:

Create a parallel AbstractJdbcBaseSource class to seperate the two hierarchies. This will inform what we can clean up and is common.
Cleans up the non-JDBC AbstractNonJdbcSource by removing cruft that is only relevant for JDBC classes

Future work:

Flatten the new AbstractJdbcBaseSource and existing AbstractJdbcSource. I suspect that we will be able to: (i) Remove a lot of redundant code. (ii) Create common util classes for shared code (e.g. constructing and decorating the read iterators)
Use the work in step 1 to inform a future interface looks like

Why this intermediate stage? Since there is some duplicate code across the two forked abstract classes

Reduces risk
Breaks inheritance between non-JDBC (in this case just mongoDB and bigquery) classes and JDBC subclasses and simplified the non-JDBC code significantly

I expect this to be a transient stage

Created a pure fork. Will spend time cleaning it up

akashkulk · 2023-03-16T23:08:44Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4442450082
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4442450082
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:509: The previous and actual discovered catalogs are identical.
=================== 62 passed, 5 skipped in 82.65s (0:01:22) ===================

akashkulk · 2023-03-16T23:08:51Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/4442450772
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/4442450772
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
================= 19 passed, 6 skipped, 27 warnings in 16.82s ==================

akashkulk · 2023-03-16T23:09:12Z

/test connector=connectors/source-bigquery

🕑 connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/4442452266
✅ connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/4442452266
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:509: The previous and actual discovered catalogs are identical.
======================== 32 passed, 2 skipped in 41.86s ========================

akashkulk · 2023-03-16T23:09:29Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/4442454510
❌ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/4442454510
🐛 https://gradle.com/s/3mnwjgugkaqbw

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestConnection::test_check[inputs0] - AssertionError: as...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs0] - doc...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs0]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
ERROR test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contain...
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
============= 2 failed, 21 passed, 1 skipped, 10 errors in 31.65s ==============

github-actions · 2023-03-16T23:11:50Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (25)

Connector	Version	Changelog	Publish
`source-alloydb`	`2.0.2`	✅	✅
`source-alloydb-strict-encrypt`	`2.0.2`	🔵 (ignored)	🔵 (ignored)
`source-bigquery`	`0.2.3`	✅	✅
`source-clickhouse`	`0.1.16`	✅	✅
`source-clickhouse-strict-encrypt`	`0.1.16`	🔵 (ignored)	🔵 (ignored)
`source-cockroachdb`	`0.1.21`	✅	✅
`source-cockroachdb-strict-encrypt`	`0.1.21`	🔵 (ignored)	🔵 (ignored)
`source-db2`	`0.1.18`	✅	✅
`source-db2-strict-encrypt`	`0.1.18`	🔵 (ignored)	🔵 (ignored)
`source-dynamodb`	`0.1.2`	✅	✅
`source-jdbc`	`0.3.5`	🔵 (ignored)	🔵 (ignored)
`source-mongodb-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`source-mongodb-v2`	`0.1.19`	✅	✅
`source-mssql`	`1.0.3`	✅	✅
`source-mssql-strict-encrypt`	`1.0.3`	🔵 (ignored)	🔵 (ignored)
`source-mysql`	`2.0.3`	✅	✅
`source-mysql-strict-encrypt`	`2.0.3`	🔵 (ignored)	🔵 (ignored)
`source-oracle`	`0.3.23`	✅	✅
`source-oracle-strict-encrypt`	`0.3.23`	🔵 (ignored)	🔵 (ignored)
`source-postgres`	`2.0.3`	✅	✅
`source-postgres-strict-encrypt`	`2.0.3`	🔵 (ignored)	🔵 (ignored)
`source-redshift`	`0.3.16`	✅	✅
`source-scaffold-java-jdbc`	`0.1.0`	🔵 (ignored)	🔵 (ignored)
`source-snowflake`	`0.1.31`	✅	✅
`source-tidb`	`0.2.3`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

akashkulk · 2023-03-17T21:31:38Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4451754450
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4451754450
🐛 https://gradle.com/s/tgxj7e472umv4

Build Failed

Test summary info:

Could not find result summary

akashkulk and others added 5 commits March 16, 2023 12:40

Some more cleanup in abstract classes in preparation of splitting

aef6eaf

Merge branch 'master' into more_refac

9dd8491

Merge branch 'master' into more_refac

5559547

Create a fork AbstractJdbcBaseSource -> AbstractDbSource

692d551

Created a pure fork. Will spend time cleaning it up

Cleanup non-JDBC hierarchy

4cf4fa9

octavia-squidington-iii added area/connectors Connector related issues connectors/source/bigquery connectors/source/mongodb-v2 connectors/source/jdbc connectors/source/relational-db labels Mar 16, 2023

Merge branch 'master' into more_refac

c22c0d0

akashkulk requested review from subodh1810, rodireich, nguyenaiden and sergio-ropero March 17, 2023 00:02

Merge branch 'master' into more_refac

4b7c573

Automated Change

9f788e4

akashkulk removed request for rodireich, nguyenaiden, subodh1810 and sergio-ropero March 22, 2023 15:03

natikgadzhi closed this Aug 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a parallel abstract class hierarchy for JDBC vs non-JDBC classes #24172

Create a parallel abstract class hierarchy for JDBC vs non-JDBC classes #24172

akashkulk commented Mar 16, 2023 •

edited

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Mar 16, 2023 •

edited

Loading

✅ Sources (25)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

akashkulk commented Mar 17, 2023 •

edited by github-actions bot

Loading

Create a parallel abstract class hierarchy for JDBC vs non-JDBC classes #24172

Create a parallel abstract class hierarchy for JDBC vs non-JDBC classes #24172

Conversation

akashkulk commented Mar 16, 2023 • edited Loading

akashkulk commented Mar 16, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Mar 16, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Mar 16, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Mar 16, 2023 • edited by github-actions bot Loading

Build Failed

github-actions bot commented Mar 16, 2023 • edited Loading

Affected Connector Report

✅ Sources (25)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

akashkulk commented Mar 17, 2023 • edited by github-actions bot Loading

Build Failed

akashkulk commented Mar 16, 2023 •

edited

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Mar 16, 2023 •

edited

Loading

akashkulk commented Mar 17, 2023 •

edited by github-actions bot

Loading